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EXECUTIVE  SUMMARY 


1.0  Introduction 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  program  was 
chartered  by  the  Deputy  Director,  Test,  Systems  Engineering  and  Evaluation  (Test  and 
Evaluation),  Office  of  the  Secretary  of  Defense  (Acquisition  and  Technology)  in  October  1994  to 
investigate  the  utility  of  advanced  distributed  simulation  (ADS)  technologies  for  support  of  test 
and  evaluation  (T&E).  The  JADS  Joint  Test  Force  (JTF)  is  Air  Force  led,  with  Army  and  Navy 
participation,  and  is  scheduled  for  completion  in  March  2000.  JADS  JTF  conducted  three  tests  of 
currently  fielded  systems  using  an  ADS-enhanced  test  environment.  The  System  Integration  Test 
(SIT)  explored  ADS  support  of  air-to-air  missile  testing;  the  End-to-End  (ETE)  Test  investigated 
ADS  support  for  command,  control,  communications,  computers,  intelligence,  surveillance  and 
reconnaissance  (C4ISR)  testing;  and  the  Electronic  Warfare  (EW)  Test  examined  ADS  support 
for  EW  testing.  The  JADS  JTF  was  also  chartered  to  observe  or  to  participate  at  a  modest  level 
in  ADS  activities  sponsored  and  conducted  by  other  agencies  in  an  effort  to  broaden  conclusions 
developed  in  the  three  dedicated  test  areas. 

Verification,  validation  and  accreditation  (W&A)  are  key  methodologies  required  for  the  use  of 
ADS  in  support  of  developmental  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation 
(OT&E).  In  keeping  with  the  JADS  charter  to  explore  the  critical  constraints,  concerns,  and 
methodologies  when  using  ADS  for  T&E,  this  report  addresses  the  present  state  of  the  W&A  of 
ADS-enhanced  test  environments.  It  first  assesses  the  status  of  the  W&A  of  ADS-enhanced  test 
environments  as  represented  by  directives,  regulations,  and  formal  processes.  Next  it  presents  the 
W&A  processes  used  by  each  of  the  three  JADS  tests,  lessons  learned,  and  finally,  the  report 
draws  conclusions  and  provides  a  recommended  guide  to  the  W&A  of  an  ADS-enhanced  test 
environment. 

2.0  Existing  W&A  Guidelines 

Currently  there  are  three  references  available,  or  in  the  process  of  being  published,  which  are  of 
extreme  importance  to  those  interested  in  the  W&A  of  ADS-enhanced  test  environments.  The 
first  of  these  documents,  the  Department  of  Defense  (DoD)  Verification,  Validation,  and 
Accreditation  (W&A)  Recommended  Practices  Guide ,  describes  an  overall  methodology  for  the 
conduct  of  W&A.  The  remaining  two  documents,  the  Recommended  Practice  for  Distributed 
Interactive  Simulation-Verification,  Validation,  and  Accreditation  and  the  High  Level 
Architecture  (HLA)  Federation  Development  and  Execution  Process  (FEDEP):  W&A  Overlay, 
describe  the  W&A  methodologies  used  for  the  two  most  prominent  forms  of  distributed 
simulation  currently  practiced.  These  directives,  regulations,  and  the  formal  processes  associated 
with  them  that  apply  to  the  W&A  of  an  ADS-enhanced  test  environment,  while  differing  in  then- 
level  of  detail,  all  require  the  W&A  of  an  ADS-enhanced  test  environment  when  used  to  test  a 
military  system. 
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3.0  W&A  of  the  Three  JADS  Tests 

The  System  Integration  Test  (SIT)  and  Electronic  Warfare  (EW)  Test  were  primarily  hardware- 
in-the-loop  tests,  or  used  hardware-in-the-loop  simulations  in  place  of  actual  hardware.  Both 
used  existing  facilities  with  well-documented,  verified  and  validated  simulations.  The  End-to-End 
(ETE)  Test,  however,  modified  a  large-scale  battlefield  simulation  and  designed  and  developed  a 
simulation  of  the  major  component  of  its  ADS  test  environment.  All  three  tests  were  different  in 
both  the  way  they  developed  the  ADS-enhanced  test  environment  and  the  way  they  applied  the 
W&A  process.  The  SIT  and  EW  Test  were  primarily  concerned  with  the  verification  and 
validation  (V&V)  of  the  environment.  The  nodes  were  already  verified  and  validated.  The  ETE 
Test,  on  the  other  hand,  had  to  first  V&V  the  nodes  and  then  V&V  the  environment.  Several 
lessons  were  learned  during  the  W&A  of  the  three  JADS  tests.  The  two  most  important  lessons 
learned  were  that  V&V  are  affordable  if  performed  in  conjunction  with  test  design,  integration, 
and  functionality  testing,  and  that  W&A  reduce  risk  for  the  tester.  The  W&A  of  an  ADS- 
enhanced  test  environment  should  proceed  hand-in-hand  with  the  design  and  development  of  the 
environment.  The  primary  role  of  W&A  is  to  reduce  risk.  If  it  doesn’t,  it  probably  isn’t  needed. 

4.0  Guide  to  the  W&A  of  an  ADS-Enhanced  Test  Environment 

Each  ADS-enhanced  test  must  tailor  the  specifics  of  the  W&A  process  to  the  needs  and 
requirements  of  the  test.  W&A  must  start  at  the  beginning  of  the  test  process  and  the  majority 
of  the  effort  should  be  concentrated  in  high-risk  areas.  This  report  proposes  a  guide  to  the 
W&A  of  an  ADS-enhanced  test  environment  that  associates  W&A  activities  with  a  JADS- 
developed  ADS  test  planning  methodology.  This  test  planning  methodology  is  proposed  in  the 
JADS  Special  Report— A  Test  Planning  Methodology-From  Concept  Development  Through  Test 
Execution.  The  referenced  report  tailors  the  steps  as  outlined  in  the  High  Level  Architecture 
(HLA)  Federation  Development  and  Execution  Process  (FEDEP).  The  result  is  an  ADS- 
enhanced  test  planning  and  implementation  methodology. 

5.0  Conclusions 

W&A  of  ADS-enhanced  test  environment  when  used  to  test  a  military  system  are  required  by  the 
various  DoD  and  service  directives  and  regulations  but  doesn’t  necessarily  have  to  be  an  onerous 
burden.  A  tailored,  requirements-driven  process,  which  is  integrated  into  the  test  from  the 
beginning,  can  be  an  affordable  activity  that  reduces  risk.  In  fact,  testers  already  do  V&V.  They 
just  don’t  document  them  as  such.  These  activities  if  properly  documented  can  become  the  V&V 
that  lead  to  accreditation. 
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1.0  Introduction 


The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E)  was 
chartered  by  the  Deputy  Director,  Test,  Systems  Engineering  and  Evaluation  (Test  and 
Evaluation),  Office  of  the  Secretary  of  Defense  (OSD)  (Acquisition  and  Technology)  in  October 
1994  to  investigate  the  utility  of  advanced  distributed  simulation  (ADS)  technologies  for  support 
of  developmental  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation  (OT&E). 

The  JADS  JT&E  charter  focused  on  three  issues:  what  is  the  present  utility  of  ADS,  including 
distributed  interactive  simulation  (DIS),  for  test  and  evaluation  (T&E);  what  are  the  critical 
constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E;  and  what  are  the 
requirements  that  must  be  introduced  into  ADS  systems  if  they  are  to  support  a  more  complete 
T&E  capability  in  the  future. 

The  JADS  Joint  Test  Force  (JTF)  investigated  ADS  applications  by  considering  three  slices  of  the 
T&E  spectrum:  the  System  Integration  Test  (SIT)  explored  ADS  support  of  air-to-air  missile 
testing;  the  End-to-End  (ETE)  Test  investigated  ADS  support  for  command,  control, 
communications,  computers,  intelligence,  surveillance  and  reconnaissance  (C4ISR)  testing;  and 
the  Electronic  Warfare  (EW)  Test  examined  ADS  support  for  EW  testing.  Each  test  focused  on 
the  three  issues  listed  above.  The  JTF  was  also  chartered  to  observe  or  participate  at  a  modest 
level  in  ADS  activities  sponsored  and  conducted  by  other  agencies  in  an  effort  to  broaden 
conclusions  developed  in  the  three  dedicated  test  areas. 

One  of  the  major  issues  associated  with  the  use  of  ADS  is  the  credibility  of  the  models  and 
simulations  (M&S)  used  and  the  credibility  of  their  interactions  over  a  distributed  network. 
Credibility  is  commonly  measured  by  verification  and  validation  (V&V)  and  then  formally 
approved  as  adequate  for  use  for  a  specific  application  by  accreditation.  The  V&V  of  distributed 
simulations  are  formally  required  by  Department  of  Defense  (DoD)  Instruction  5000.61,  DoD 
Modeling  and  Simulation  Verification,  Validation  and  Accreditation ,  prior  to  their  use  in  testing. 

Verification,  validation  and  accreditation  (W&A)  are  key  methodologies  required  for  the  use  of 
ADS  in  support  of  DT&E  and  OT&E.  In  keeping  with  JADS  charter  to  explore  the  critical 
constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E,  this  report  will  assess  the 
present  state  of  the  W&A  of  ADS-enhanced  test  environments  by  first  assessing  the  formal 
status  of  the  W&A  of  ADS-enhanced  test  environments  as  represented  by  directives,  regulations, 
and  formal  processes.  It  will  then  present  the  W&A  processes  used  by  the  three  aforementioned 
tests  conducted  by  the  JADS  JTF.  Following  this  will  be  lessons  learned  as  a  result  of  conducting 
the  W&A  processes  and  a  guide  to  the  W&A  of  an  ADS-enhanced  test  environment.  Finally, 
the  report  will  provide  some  conclusions  and  recommendations. 
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2.0  Verification,  Validation  and  Accreditation 

Prior  to  any  discussion  of  W&A,  it  is  necessary  to  first  establish  a  common  frame  of  reference. 
The  definitions  that  follow  are  taken  from  the  DoD  Manual  5000.59,  DoD  Modeling  and 
Simulation  (M&S)  Glossary. 

Accreditation.  The  official  certification  that  a  model  or  simulation  is  acceptable  for  use  for  a 
specific  purpose. 

It  is  important  to  note  that  the  definition  states  that  the  accreditation  is  for  a  specific  purpose.  No 
mention  is  made  of  a  class  accreditation  or  general-purpose  accreditation.  Missing  from  the 
definition  is  the  reference  to  federations  of  models  and  simulations  present  in  the  definition  used  in 
DoD  Instruction  5000.61. 

Federation  Element.  Term  applied  to  an  individual  model  and/or  simulation  that  is  part  of  a 
federation  of  models  and  simulations. 

This  and  the  following  definition,  taken  from  DoD  Instruction  5000.61,  form  the  basis  of  what  is 
commonly  referred  to  within  JADS  as  the  synthetic  environment. 

Federation  of  Models  and  Simulations.  A  system  of  interacting  models  and/or  simulations,  with 
supporting  infrastructure,  based  on  a  common  understanding  of  the  objects  portrayed  in  the 
system. 

Model.  A  physical,  mathematical,  or  otherwise  logical  representation  of  a  system,  entity, 
phenomenon,  or  process. 

Modeling  and  Simulation.  The  use  of  models,  including  emulators,  prototypes,  simulators,  and 
stimulators,  either  statically  or  over  time,  to  develop  data  as  a  basis  for  making  managerial  or 
technical  decisions.  The  terms  "modeling"  and  "simulation"  are  often  used  interchangeably. 

Or,  as  stated  in  DoD  Instruction  5000.61: 

Modeling  and  Simulation.  The  development  and  use  of  live,  virtual,  and  constructive  models 
including  simulators,  stimulators,  emulators,  and  prototypes  to  investigate,  understand,  or 
provide  experimental  stimulus  to  either  (1)  conceptual  systems  that  do  not  exist  or  (2)  real  life 
systems  which  cannot  accept  experimentation  or  observation  because  of  resource,  range, 
security,  or  safety  limitations.  This  investigation  and  understanding  in  a  synthetic  environment 
will  support  decisions  in  the  domains  of  research,  development,  and  acquisition  and  analysis,  or 
transfer  necessary  experimental  effects  in  the  education,  training,  and  military  operations 
domain. 

Within  JADS,  M&S,  when  distributed,  are  commonly  referred  to  as  ADS. 
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Simulation.  A  method  for  implementing  a  model  over  time. 


Validation.  The  process  of  determining  the  degree  to  which  a  model  or  simulation  is  an  accurate 
representation  of  the  real  world  from  the  perspective  of  the  intended  uses  of  the  model  or 
simulation. 

The  perspective  of  the  intended  uses  of  a  model  is  normally  expressed  in  terms  of  requirements 
for  the  model.  This  provides  a  basis  for  determining  the  degree  to  which  a  model  is  an  accurate 
representation. 

Verification.  The  process  of  determining  that  a  model  or  simulation  implementation  accurately 
represents  the  developer’s  conceptual  description  and  specification.  Verification  also  evaluates 
the  extent  to  which  the  model  or  simulation  has  been  developed  using  sound  and  established 
software  engineering  techniques. 

Again,  the  developer’s  conceptual  description  and  specifications  are  expressed  as  requirements  for 
the  model.  The  latter  sentence  in  this  definition  is  felt  to  apply  more  to  software  verification, 
validation  and  certification  (W&C). 

The  definitions  given  are  deemed  important  for  the  purposes  of  this  report.  It  is  recommended 
that  the  reader  review  all  the  definitions  given  in  DoD  Manual  5000.59  for  a  more  complete 
understanding  of  W&A. 

2.1  DoD  Regulation  5000.2 

DoD  Regulation  5000.2,  Mandatory  Procedures  for  Major  Defense  Acquisition  Programs 
(MDAPS)  and  Major  Automated  Information  System  (MAIS)  Acquisition  Programs,  is  important 
in  the  consideration  of  the  W&A  of  ADS-enhanced  test  environments.  This  regulation 
authorizes  the  use  of  accredited  modeling  and  simulation,  as  appropriate,  throughout  a  system’s 
life  cycle  in  support  of  the  various  acquisition  activities  including  test  and  evaluation.  The 
requirement  that  accredited  models  and  simulations  be  used  implies  that  the  appropriate 
verification  and  validation  has  been  performed. 

2.2  DoD  Directive  5000.59 

DoD  Directive  5000.59,  DoD  Modeling  and  Simulation  (M&S)  Management,  establishes  DoD 
policy,  assigns  responsibilities,  and  prescribes  procedures  for  the  management  of  models  and 
simulations.  Key  established  W&A  policies  are  the  requirements  for  the  DoD  components  to 
establish  verification,  validation,  and  accreditation  policies  and  procedures  for  M&S  applications 
managed  by  the  DoD  component,  and  M&S  applications  used  to  support  the  major  DoD  decision 
making  organizations  will  be  accredited. 
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2.3  DoD  Instruction  5000.61 


DoD  Instruction  5000.61,  DoD  Modeling  and  Simulation  (M&S)  Verification,  Validation,  and 
Accreditation ,  implements  policy,  assigns  responsibilities,  and  prescribes  procedures  for  the 
W&A  of  DoD  M&S. 

Of  particular  importance  is  the  policy  implemented  within  DoD  5000.61.  Specifically,  the 
instruction  states 

1 .  DoD  components  shall  establish  W&A  policies  and  procedures  for  M&S  they 
develop  and  manage.  The  “DoD  M&S  executive  agent”  (MSEA)  responsible  for 
a  common  or  general-use  M&S  shall  follow  DoD  component  W&A  policies 
and  procedures  for  that  application,  unless  otherwise  specified  by  the  Under 
Secretary  of  Defense  for  Acquisition  and  Technology  (USD(A&T)). 

2.  M&S  used  to  support  the  major  DoD  decision  making  organizations  and 
processes  (such  as  the  Defense  Planning  and  Resources  Board;  the  Defense 
Acquisition  Board  (DAB);  the  Joint  Requirements  Oversight  Council;  and  the 
DoD  Planning,  Programming,  and  Budgeting  System  (described  in  references  (b) 
through  (f)))  shall  be  accredited  for  that  use  by  the  DoD  component  sponsoring 
the  application.  Likewise,  M&S  used  for  joint  training  and  joint  exercises  shall  be 
accredited  for  that  purpose  by  the  application  sponsor. 

3.  Each  DoD  component  shall  be  the  final  authority  for  validating  representations 
of  its  own  forces  and  capabilities  in  joint-,  general-,  and  common-use  M&S 
applications  and  shall  be  responsive  to  the  other  DoD  components  to  ensure  they 
are  appropriately  represented. 

In  addition,  the  USD  (A&T)  will,  through  the  DoD  Executive  Council  on  Modeling  and 
Simulation,  designate  the  Defense  Modeling  and  Simulation  Organization  (DMSO)  as  the  DoD 
focal  point  to  be  the  central  source  of  DoD  W&A  data  and  information  and  to  assist  DoD  and 
non-DoD  organizations,  as  requested,  in  resolving  W&A  issues  or  obtaining  information  on 
DoD  W&A  activities. 

Key  among  the  procedures  stated  in  the  instruction  is  that  V&V  will  be  required  after  the  effective 
date  of  the  instruction  for  all  major  M&S  developments.  Major  M&S  include,  but  are  not  limited 
to,  M&S  whose  intended  application  will  require  accreditation  by  DoD  or  component  policy;  that 
will  be  elements  of  a  federation  of  models  and  simulations;  that  are  intended  for  reuse;  whose 
application  involves  safety  of  life;  and  whose  development  will  involve  the  commitment  of 
significant  DoD  resources.  V&V  are  also  required  when  existing  models  and  simulations  undergo 
major  upgrade  or  modification  and  will  be  incorporated  into  the  development  and  life-cycle 
management  processes  for  all  major  M&S.  V&V  are  encouraged  for  all  major  M&S  currently  in 
use  in  the  DoD. 
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The  requirement  to  V&V  M&S  developments  that  require  accreditation  establishes  the  need  to 
V&V  federations  or  synthetic  environments  that  will  be  used  in  the  acquisition  process  including 
test  and  evaluation. 

2.4  Service  Regulations 

2.4.1  Army  Policies,  Procedures,  and  Guidelines 

Army  Regulation  (AR)  5-11,  Management  of  Army  Models  and  Simulations,  prescribes  policy 
and  guidance  and  assigns  responsibilities  for  the  management  of  Army  models  and  simulations. 
Included  in  this  regulation  is  the  Army  policy  on  W&A. 

Of  interest  to  Army  testers  is  the  requirement  that  “W&A  activities  will  be  included  in  the 
simulation  support  plan  required  for  all  Army  acquisition  strategies  for  Acquisition  Category  I  and 
II  and  non-major  programs.  A  summary  of  the  W&A  status  of  M&S  used  or  proposed  for  use 
to  support  test  and  evaluation  for  an  acquisition  program,  to  include  resources  required,  will  be 
included  in  the  program's  Test  and  Evaluation  Master  Plan  (see  DoD  5000.2-R).” 

In  addition,  “W&A  for  a  federation  of  M&S  will  adhere  to  not  only  the  general  W&A  policies 
and  procedures  for  each  individual  M&S,  but  will  also  consider  system  compliance,  compatibility, 
and  interoperability  requirements.  W&A  of  a  federation  will  ensure  credible  results  of  the 
integrated  system  as  a  whole.  Each  unique  application  configuration  of  a  federation  requires 
W&A.” 

With  regard  to  accreditation,  the  regulation  states  that  “M&S  used  to  support  major  DoD 
decision  making  organizations  and  processes  (such  as  the  Defense  Planning  and  Resources  Board 
(DPRB);  the  Defense  Acquisition  Board  (DAB);  the  Joint  Requirements  Oversight  Council 
(JROC);  and  the  DoD  Planning,  Programming,  and  Budgeting  System  (PPBS))  will  be  accredited 
specifically  for  that  usage  by  the  applicable  Army  application  sponsor.” 

It  goes  on  to  say  “Accreditation  is  a  management  responsibility  of  the  application  sponsor, 
assisted  by  the  V&V  agent.  The  specific  use  of  the  M&S  must  be  considered  in  context  with  its 
capabilities  and  limitations.  Accreditation  includes  data  inputs,  scenarios,  and  the  operators- 
analysts-trainers  who  will  use  the  M&S.” 

The  Army  regulation  does  provide  for  the  accreditation  of  M&S  for  a  generic  set  of  applications. 
However,  each  specific  usage  of  that  M&S  still  requires  accreditation  by  the  specific  application 
sponsor. 

Department  of  the  Army  Pamphlet  (DA  PAM)  5-11,  Verification,  Validation,  and  Accreditation 
of  Army  Models  and  Simulations,  gives  procedures  for  the  "Army  Model  and  Simulation 
Management  Program"  (AR  5-11).  The  objective  of  this  pamphlet  is  to  assist  the  models  and 
simulations  developer,  proponent,  and  application  sponsor  in  conforming  to  the  W&A  policies 
prescribed  in  AR  5-11.  This  pamphlet  also  provides  guidance  for  the  development,  execution,  and 
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reporting  of  all  W&A  activities.  This  pamphlet  does  not  cover  data  certification  procedures,  but 
it  does  address  data  certification  in  reference  to  proper  M&S  use. 

DA  PAM  5-11  is  currently  under  revision  to  update  procedures  for  the  Army  Model  and 
Simulation  Management  Program.  It  will  provide  new  guidance  for  compliance  with  the  high 
level  architecture  (HLA)  and  revised  instructions  for  the  development,  execution,  and  reporting  of 
all  W&A  activities. 

2.4.2  Navy  Policies,  Procedures,  and  Guidelines 

The  Secretary  of  the  Navy  Instruction  (SECNAVINST)  5200.40,  Verification,  Validation,  and 
Accreditation  (W&A)  of  Models  and  Simulations,  is  the  Navy’s  implementation  of  DoD 
Directive  5000.59  and  DoD  Instruction  5000.61  that  requires  the  DoD  components  to  establish 
W&A  policies,  procedures,  and  guidelines  for  M&S  applications.  It  applies  to  both  Navy  and 
Marine  Corps  activities. 

The  instruction  states  that  M&S  activities  that  are  “...used  significantly  in  supporting  the 
development  of  either  the  Department  of  the  Navy  (DON)  program  objectives  memorandum 
(POM)  or  analysis  of  alternatives”  will  be  verified  and  validated,  as  set  forth  in  the  instruction, 
prior  to  their  use.  The  instruction  also  requires  that  these  M&S  activities  must  be  accredited  by 
the  appropriate  accreditation  authority  prior  to  using  them  in  support  of  the  above  applications.  It 
does  allow,  however,  the  accreditation  authority  to  tailor  the  W&A  process  to  meet  specific 
requirements  and  objectives  while  working  within  existing  resource  constraints.  The  accreditation 
authority  is  specified  in  SECNAVINST  5000.2B,  Implementation  of  Mandatory  Procedures  for 
Major  and  Non-Major  Defense  Acquisition  Programs  and  Major  and  Non-Major  Information 
Technology  Acquisition  Programs. 

The  instruction  also  contains  a  description  of  the  Navy’s  M&S  development  process  with  key 
elements  of  the  W&A  process  emphasized  throughout. 

2.4.3  Air  Force  Policies,  Procedures,  and  Guidelines 

Air  Force  Instruction  16-1001,  Verification,  Validation  and  Accreditation  (W&A),  requires  that 
all  Air  Force  M&S  applications  supporting  the  major  DoD  decision  making  organizations  or 
processes  specified  in  DoD  5000.59  (paragraph  D.8)  and  Air  Force  input  to  these  DoD 
organizations  will  be  accredited  for  their  intended  purpose.  The  instruction  further  states  that  all 
executed  V&V  activities  will  support  the  model  acceptance/accreditation  requirements  defined  by 
the  accreditation  authority. 

Like  the  previous  service  policies,  procedures,  and  guidelines,  this  instruction  also  contains  a 
W&A  process.  Unlike  the  other  services,  however,  this  process  does  not  appear  to  be  tied  to 
the  M&S  requirements  but  rather  to  accreditation  requirements  established  by  the  accreditation 
authority.  The  minimum  accreditation  requirements  cited  appear  to  focus  more  heavily  on  M&S 
development  history,  past  applications,  configuration  control  and  similar  requirements.  To  be  fair. 
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however,  the  instruction  does  require  the  M&S  capabilities  and  credibility,  based  on  Y&V  status 
to  be  compared  to  the  application  criteria. 

2.5  Assessment  of  Current  ADS  W&A  Methodologies 

There  are  currently  three  references  available,  or  in  the  process  of  being  published,  that  are  of 
extreme  importance  to  those  interested  in  the  use  of  ADS-enhanced  test  environments.  The  first 
of  these  documents,  the  Department  of  Defense  (DoD)  Verification,  Validation,  and 
Accreditation  (W&A)  Recommended  Practices  Guide,  describes  an  overall  methodology  for  the 
conduct  of  W&A.  The  remaining  two  documents,  the  Recommended  Practice  for  Distributed 
Interactive  Simulation  —Verification,  Validation,  and  Accreditation  and  the  High  Level 
Architecture  (HLA)  Federation  Development  and  Execution  Process  (FEDEP):  W&A  Overlay, 
describe  the  W&A  methodologies  used  for  the  two  most  prominent  forms  of  distributed 
simulation  currently  practiced. 

2.5.1  Department  of  Defense  Verification,  Validation,  and  Accreditation  Recommended 
Practices  Guide 

The  DoD  W&A  RPG  is  an  outstanding  guide  that  contains  something  for  every  developer  and 
user  in  DoD  involved  in  the  W&A  of  an  ADS-enhanced  test  environment. 

It  contains  sections  that  provide  to  the  decision  makers  a  quick  overview  of  W&A  and 
information  on  the  accreditation  process.  It  also  provides  program  managers  a  quick  overview  of 
what  W&A  are  all  about  as  a  necessary  introduction.  Program  managers  will  also  be  interested 
in  the  principles  and  processes  of  W&A  as  they  incorporate  these  into  their  programs.  Finally,  it 
assists  program  managers  in  preparing  senior  decision  makers  for  the  accreditation  decision. 

The  technical  staff,  whose  job  is  to  do  the  actual  V&V,  should  read  the  entire  document.  In 
addition  to  the  sections  noted  above,  it  will  give  users  valuable  guidance  on  specific  techniques 
that  are  used  in  V&V  and  will  provide  common  reporting  formats  to  help  them  document  the 
W&A  effort.  The  current  DoD  W&A  RPG  is  available  at  the  DMSO  website 

(www.DMSO.mil). 

The  DoD  W&A  RPG  is  currently  under  revision  and  will  soon  be  released  for  general  use.  The 
revision  will  incorporate  the  use  of  web  technology  to  allow  the  user  to  navigate  a  focused  path 
based  upon  the  user’s  function  with  respect  to  W&A.  The  program  manager  may  follow  a 
thread  that  addresses  W&A  at  the  program  level  or  may  follow  specific  links  for  more  detailed 
information.  The  practitioner  can  immediately  access  detailed  information  on  how  to  conduct 
W&A.  The  revised  RPG  is  scheduled  to  be  available  on  the  DMSO  web  site  in  March  2000. 
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2.5.2  Recommended  Practice  for  Distributed  Interactive  Simulation  --Verification, 
Validation,  and  Accreditation 

The  Recommended  Practice  for  Distributed  Interactive  Simulation— Verification,  Validation,  and 
Accreditation  (DIS  RPG)  establishes  guidelines  for  the  W&A  of  DIS  exercises  and  includes  the 
use  of  DIS-enhanced  test  environments.  It  provides  “how  to”  procedures  for  planning  and 
conducting  DIS  exercise  W&A. 

The  DIS  RPG  is  intended  for  use  by  persons  responsible  for  or  participating  in  DIS-enhanced  test 
environment  W&A  activities  (e.g.,  W&A  agents,  W&A  teams,  test  managers,  test  architects). 
It  also  provides  W&A  guidance  to  enhanced  test  environment  users/sponsors  and  developers. 

The  DIS  RPG  (Institute  of  Electrical  and  Electronics  Engineers  [IEEE]  1278-4)  is  part  of  a  set  of 
IEEE  standards  (IEEE  1278)  and  supporting  documentation  that  includes  standards  for 
application  protocol  and  communication  services;  profiles  standards  to  support  DIS 
interoperability;  a  recommended  practice  for  exercise  management  and  feedback  that  provides 
guidelines  for  setting  up  and  conducting  a  DIS  exercise. 

2.5.3  HLA  W&A  FEDEP  Overlay  Process 

An  RPG  for  high  level  architecture  (HLA)  W&A  has  not  yet  been  published  but  is  presently 
being  worked.  The  HLA  W&A  process  has  been  developed  and  mapped  to  the  HLA 
development  process,  known  as  the  federation  development  and  execution  process  (FEDEP) 
model.  The  mapping  of  the  W&A  process  to  the  FEDEP  allows  the  user  of  an  enhanced  ADS 
test  environment  to  tailor  the  W&A  effort  to  the  test.  Information  about  the  HLA  W&V 
FEDEP  overlay  process  is  available  from  DMSO. 

Once  the  W&A  effort  is  tailored,  the  DoD  W&A  RPG  can  be  used  to  provide  decision  makers, 
program  managers,  and  technical  staff  valuable  guidance  on  the  W&A  process  until  such  time  as 
the  HLA  W&A  RPG  is  published. 
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3.0  W&A’s  Role  in  JADS  JTF  Distributed  T&E 

As  mentioned  previously,  the  JADS  JTF  conducted  three  tests  of  currently  fielded  systems  using 
an  ADS-enhanced  test  environment.  The  SIT  and  EW  Test  were  primarily  hardware-in-the-loop 
(HWIL)  tests,  or  used  hardware-in-the-loop  simulations  in  place  of  the  actual  hardware.  Both 
utilized  existing  facilities  with  well-documented,  verified  and  validated  simulations.  The  ETE 
Test,  however,  was  required  to  develop,  or  modify,  large-scale  battlefield  simulations  and  had  to 
design  and  develop  a  simulation  of  the  major  component  of  its  ADS-enhanced  test  environment. 
All  three  tests  were  different  in  both  the  way  they  developed  the  ADS-enhanced  test  environment 
and  the  way  they  applied  the  W&A  process.  The  SIT  and  EW  Test  were  primarily  concerned 
with  the  verification  and  validation  of  the  environment.  The  nodes  were  already  verified  and 
validated.  The  ETE  Test,  on  the  other  hand,  had  to  first  V&V  the  nodes  and  then  V&V  the 
environment. 

3.1  System  Integration  Test  W&A 

A  major  objective  of  the  SIT  was  to  evaluate  the  validity  of  the  missile  performance  data  obtained 
using  the  ADS  configurations.  V&V  methods  developed  for  this  purpose  were  consistent  with 
those  in  the  DoD  Verification,  Validation  and  Accreditation  (W &A)  Recommended  Practices 
Guide,  specifically  bottom-up  testing,  functional  testing,  data  integration  testing,  predictive 
validation,  and  comparison  testing. 

The  missile  HWIL  simulations  had  been  previously  verified  for  their  intended  uses.  Therefore,  the 
distributed  test  verification  was  to  confirm  that  the  ADS  configuration  operated  as  designed  when 
the  nodes  were  real  time  linked  through  an  ADS  network  and  could  be  used  to  replicate  an  actual 
live  fire  test  including  the  necessary  data  collection.  Verification  was  an  incremental  and  iterative 
process  beginning  pretest  and  continuing  throughout  the  missions.  The  structured  approach  began 
with  systematic  check-out  of  each  stand-alone  simulator/facility,  followed  by  testing  of  each  pair 
of  linked  nodes,  and  culminated  with  the  check-out  of  the  complete  linked  architecture  consistent 
with  the  bottom-up  testing  method  in  the  DoD  RPG. 

The  objective  of  the  validation  process  was  to  demonstrate  that  the  performance  of  the  missile  in 
the  HWIL  lab,  driven  by  data  from  the  aircraft,  was  consistent  with  that  of  an  actual  missile  live 
fire  profile  under  the  same  conditions. 

The  preferred  method  for  validation  would  be  to  replicate  previous  live  fire  test  profiles  using  the 
ADS  configuration  and  to  directly  compare  results  using  quantitative  techniques.  However, 
several  complications  prevented  this  direct  comparison. 

Because  of  these  complications,  the  validation  method  employed  used  a  multistep  process. 

-  The  validity  of  the  missile  HWIL  simulation  itself  was  established  by  comparison  to  live  fire 
data  consistent  with  the  predictive  validation  method  in  the  DoD  RPG. 
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-  The  validity  of  each  SIT  linked  run  was  judged  by  missile  experts  based  on  a  qualitative 
assessment  of  whether  the  missile  HWIL  simulation  responded  properly  to  the  inputs  received 
during  the  run. 

-  The  validity  of  selected  SIT  linked  runs  was  established  by  comparison  to  HWIL  lab  stand¬ 
alone  runs,  consistent  with  the  comparison  testing  method  in  the  DoD  RPG.  NOTE:  In 
this  case,  the  missile  HWIL  simulation  was  employed  as  a  truth  source  for  data  comparison 
purposes,  since  it  was  a  validated  representation  of  the  actual  missile  (as  established  in  the 
first  step  above). 

This  approach  compared  statistical  distributions  of  significant  performance  parameters,  affording 
a  higher  level  of  confidence  in  the  comparisons.  In  addition,  this  technique  allowed  for  the 
validation  of  profiles  for  which  no  live  data  existed. 

3.2  End-To-End  Test  W&A 

The  ETE  Test  was  designed  to  evaluate  the  utility  of  ADS  to  support  testing  of  C4ISR  systems. 
The  test  focused  on  the  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS)  as  one 
component  of  a  representative  C4ISR  system.  The  ETE  Test  also  evaluated  the  capability  of  the 
JADS  Test  Control  and  Analysis  Center  (TCAC)  to  control  a  distributed  test  of  this  type  and 
remotely  monitor  and  analyze  test  results. 

Since  JADS  is  a  DoD-sponsored  joint  test,  the  overall  guide  for  the  V&V  of  the  JADS  End-to- 
End  (ETE)  Test  was  the  DoD  W&A  RPG.  The  JADS  ETE  Test  utilized  the  IEEE  Standard 
1278  for  DIS  to  develop  its  ADS-enhanced  test  environment.  The  DoD  W&A  RPG  advocated 
the  use  of  the  DIS  nine  step  W&A  process  for  the  W&A  of  the  DIS-enhanced  test 
environment. 

The  DIS  nine  step  W&A  process  was  developed  with  the  knowledge  that  it  must  be  tailored  to 
each  individual  use.  The  tailoring  of  the  W&A  process  resulted  in  the  JADS  ETE  Test  W&A 
process  model  shown  in  Figure  1. 
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Conduct 

Test 


Numbers  correspond  to  steps  in  DIS  Nine  Step  W&A 


Figure  1.  JADS  ETE  Test  W&A  Process  Model 

In  the  JADS  ETE  Test  W&A  process  model,  test  events  consisting  of  planning,  constructing  and 
assembling  the  synthetic  environment  (SE);  integrating  and  testing  the  SE;  accrediting  the  SE;  and 
conducting  the  test  all  proceed  on  the  left  side  from  top  to  bottom.  The  V&V  events  including 
documentation  proceed  to  the  right  for  each  major  test  event. 

The  major  change  from  the  DIS  nine  step  W&A  process  was  the  inclusion  of  the  ADS-enhanced 
test  environment  accreditation  as  a  part  of  the  test  process,  resulting  in  an  eight  step  V&V 
process.  Accreditation  is  not  a  part  of  the  DIS  exercise  process  in  the  DIS  VV&A  process 
model.  Instead,  it  is  shown  as  a  part  of  the  W&A  process  and  by  implication  is  not  something 
that  is  required  prior  to  conducting  the  DIS  exercise.  When  ADS  is  used  in  support  of 
operational  and  developmental  testing,  accreditation  of  M&S  is  a  management  function  and  is  a 
mandatory  part  of  the  test  process. 


The  V&V  of  the  ETE  Test  ADS-enhanced  environment  were  based  upon  the  functional 
requirements  and  acceptability  criteria  that  were  taken  from  the  test  plans  and  other  documents 
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that  describe  the  test  environment.  The  V&V  agent  developed  the  V&V  plan  by  identifying  the 
tasks  required  to  satisfy  these  requirements  and  the  acceptability  criteria  in  a  manner  that  matched 
and  complemented  the  test  activity  plan,  test  requirements,  component  requirements,  available 
resources,  and  timelines. 

3.3  Electronic  Warfare  Test  W&A 

A  major  objective  of  the  EW  Test  was  to  evaluate  the  validity  of  the  ADS-produced  engagements 
when  compared  to  the  open  air  range  (OAR)  and  hardware-in-the-loop  (HITL)  representations  of 
the  same  engagements.  The  EW  Test  team  had  two  opportunities  to  accredit  the  Phase  2  test 
environment.  The  first  opportunity  occurred  pretest  and  used  the  results  of  the  formal  acceptance 
and  integration  testing  JADS  accomplished  as  part  of  the  test  preparation  and  rehearsal.  The 
second  opportunity  used  the  correlation  of  the  ADS-based  testing  with  the  results  from  the  OAR 
test  and  from  the  HITL  test  to  determine  if  the  results  from  the  federation  were  valid.  The  efforts 
undertaken  to  accredit  the  federation  are  described  below. 

3.3.1  Phase  2  V&V 

The  results  of  the  Phase  2  federate  acceptance  test  (FAT)  and  federation  integration  tests  (FIT) 
were  presented  to  the  accreditation  board  prior  to  the  JADS  test  readiness  review.  The  team 
reported  that  in  three  instances  the  aircraft  appeared  to  hover.  The  cause  could  not  be  established 
and  the  board  was  concerned  that  other  issues  might  be  present  that  would  make  the  data  invalid. 
The  accreditation  authority  gave  the  EW  Test  team  a  chance  to  remedy  or  address  the  hovering 
aircraft  issue  and  look  for  other  issues  before  executing  Phase  2  as  scheduled.  The  hovering 
aircraft  was  addressed  and  no  other  issues  were  found.  The  federation  was  accredited.  However, 
the  V&V  plan  had  to  be  republished  after  Phase  2  execution  to  satisfy  the  accreditation  board. 

The  EW  Test  architecture  used  several  software  components  that  were  not  models  in  the 
traditional  sense.  These  components  fit  into  two  categories.  The  first  category  was  HLA- 
compliant  software  that  acted  as  a  gateway  to  the  federation  allowing  either  a  non-HLA- 
compliant  model  or  data  collection  software  to  interface  with  the  rest  of  the  federates.  The 
critical  issues  for  this  software  were  that  data  were  not  corrupted  and  that  data  were  not  delayed 
excessively  as  they  were  passed  through  the  gateway.  The  second  category  was  HLA-compliant 
software  that  published  data  from  a  script.  In  this  software,  the  critical  issues  were  the  validity  of 
the  script  and  ensuring  the  software  did  not  corrupt  or  delay  the  data.  Neither  category  of 
software  was  independently  accredited.  The  EW  Test  team  used  an  acceptance  test  to  verify  that 
the  critical  issues  were  met.  The  acceptance  testing  supported  the  V&V  of  the  software  and  each 
component  was  accredited  as  part  of  the  federation. 

3.3.2  Phase  3  V&V 

The  Phase  3  test  environment  was  the  same  as  in  Phase  2  except  for  the  jammer  federate. 
Consequently,  for  Phase  3,  the  V&V  addressed  critical  functionality  and  fidelity  aspects  of  the 
jammer  in  the  installed  systems  test  facility  (ISTF).  V&V  addressed  three  key  data  sources:  threat 
parameters,  jammer  parameters,  and  aircraft  radar  cross  section.  Accreditation  of  the  threats 
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consisted  of  a  documentation  search  and  key  personnel  interviews  to  determine  limitations  that 
could  impact  the  JADS  effort.  During  V&V  for  Phase  2,  JADS  was  able  to  fmd  limited 
accreditation  information  on  the  Air  Force  Electronic  Warfare  Evaluation  Simulator  (AFEWES) 
threats.  Accreditation  information  was  directed  at  a  threat  baseline  established  through  the 
intelligence  community  and  documented  in  the  electronic  warfare  integrated  reprogramming 
(EWIR)  database.  W&C  were  provided  by  AFEWES  following  established  OAR  quality  control 
procedures.  On  the  receiver  side  of  the  jammer,  threat  parameters  from  AFEWES  were 
transformed  by  Advanced  Tactical  Electronic  Warfare  Environment  Simulator  (ATEWES)  radio 
frequency  (RF)  signal  generation  capabilities  (e.g.,  correct  power,  correct  RF  waveform).  W&C 
of  threat  parameters  were  performed  prior  to  the  FAT  by  Georgia  Tech  Research  Institute 
(GTRI)  and  approved.  Due  to  the  critical  functions  ATEWES  performed,  continuous  monitoring 
of  threat  parameters  was  done  during  Phase  3  execution.  Aircraft  radar  cross  section  was 
approved  by  the  EW  Test  integrated  product  team  (IPT)  and  the  W&C  were  provided  by  the 
results  of  AFEWES  quality  control  procedures.  No  issues  were  identified  with  key  data  sources 
during  Phase  3  V&V.  The  results  of  these  V&V  activities  were  reported  to  and  approved  by  the 
JADS  accreditation  board  prior  to  test  execution. 
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4.0  W&A  Lessons  Learned 


Several  lessons  were  learned,  or  relearned,  during  the  W&A  of  the  three  JADS  tests.  They  are 
presented  here  in  roughly  the  same  order  as  they  would  apply  during  the  W&A  of  a  test 
environment. 

The  primary  role  of  V&V  during  the  conduct  of  an  ADS-enhanced  test  is  to  reduce  risk  and  meet 
policy  requirements.  The  tester  must  have  a  high  degree  of  confidence  that  the  ADS-enhanced 
test  environment  will  work  prior  to  conducting  costly  and  resource  intensive  test  events.  V&V 
can  provide  that  high  degree  of  confidence.  The  greater  the  risk,  the  more  V&V  are  required. 
Low  risk  test  environments  require  only  enough  V&V  to  meet  policy  requirements. 

Requirements  must  exist  before  any  V&  V  can  be  conducted.  This  is  all  right  because  a  federation 
needs  requirements  before  it  can  be  developed.  If  the  developer  of  the  federation  cannot  state 
what  it  is  supposed  to  do,  it  is  extremely  difficult  to  build  and  V&V  the  federation.  This  holds 
true  for  both  the  overall  federation  and  the  individual  federates  that  comprise  the  federation.  The 
level  of  V&V  should  match  the  level  of  sophistication  of  the  requirements  and  their  associated 
acceptability  criteria.  When  the  requirements  and  acceptability  criteria  are  conceptual  in  nature, 
only  conceptual  V&V  can  be  conducted.  As  the  requirements  become  more  detailed  and  refined, 
the  V&V  should  become  more  detailed  and  refined. 

Involve  the  accreditation  folks  early  in  the  process.  They  are  responsible  for  developing  the 
acceptability  criteria  based  upon  the  test  requirements.  The  V&V  plan,  with  its  associated  cost 
estimate,  cannot  be  developed  until  the  acceptability  criteria  are  agreed  upon  between  the  test 
manager  and  the  accreditation  authority. 

Acceptability  criteria  are  specifications  the  federates  and  federation  must  meet.  V&V  are  the 
testing  required  to  determine  how  well  they  meet  specifications.  Requirements  must  exist  before 
acceptability  criteria  can  be  developed.  Acceptability  criteria  are  a  measure  of  the  risk  the 
accreditation  authority  is  willing  to  accept  prior  to  conducting  the  test. 

If  a  V&V  activity  does  not  reduce  risk  it  is  probably  not  needed.  The  developer  wants  to  know 
how  well  the  federates  and  federation  meet  the  requirements  in  order  to  proceed  with  minimal 
risk.  Risk  reduction  is  important  because  there  are  always  a  limited  number  of  test  events,  either 
because  of  money,  time,  or  limited  availability  of  test  assets.  Therefore,  the  tester  wants  to 
maximize  the  value  received  from  a  test  event  and  minimize  the  chance  of  something  going  wrong 
that  would  prevent  collecting  valid  system  under  test  (SUT)  data.  This  is  done  by  conducting 
V&V. 

V&V  are  affordable,  provided  they  are  integrated  into  federates  and  federation  testing.  The 
developer’s  first  thought  should  be  what  do  I  want  my  federation  to  do.  The  second  thought 
should  be  how  am  I  going  to  test  or  V&V  it.  The  V&V  process  must  be  integrated  into  the  test 
process  from  the  beginning. 
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Most  testers  already  do  V&V;  they  just  don’t  realize  and  document  them  as  such.  A  test 
developer  needs  to  know  if  the  ADS-enhanced  test  environment  provides  valid  SUT  data  prior  to 
using  it.  To  accomplish  this  a  series  of  tests  that  will  reduce  the  risk  of  proceeding  is  developed. 
This  series  of  tests,  if  documented,  can  become  the  V&V  required  prior  to  accreditation. 

Tailor  the  V&V  process.  Unneeded  V&V  kill  trees,  cost  time  and  money,  and  do  nothing  to 
reduce  risk. 

It  is  often  impossible  to  fully  validate  an  ADS-enhanced  test  environment.  This  is  because  the 
real  world  depicted  by  the  environment  is  nonexistent  or  is  not  well  documented.  An  example 
would  be  an  enemy  corps  rear  area.  We  can  depict  them  doctrinally,  but  until  they  take  to  the 
field,  we  can  not  validate  our  depiction.  Validation  of  an  ADS-enhanced  test  environment  is  often 
a  matter  of  getting  a  group  of  subject  matter  experts  (SME)  to  agree  that  your  test  environment  is 
a  valid  representation  of  reality  without  actually  knowing  what  reality  is.  Involve  the 
accreditation  authorities  in  selecting  the  SMEs  and  they  will  be  more  willing  to  accept  then- 
findings. 

No  complex  system  is  ever  used  the  way  the  designer  thought  it  would  be  used.  This  will  have  an 
impact  on  V&V.  V&V  should  be  able  to  detect  this  deviate  usage  and  determine  what  effect  it 
will  have  on  the  system  and  the  synthetic  environment. 

V&V  should  consider  breakdowns  or  faults  in  the  SUT.  In  our  zeal  to  build  perfect  simulations, 
we  often  forget  that  everything  breaks  or  malfunctions  at  some  point  in  its  usage.  Often, 
reliability  data  or  predictions  are  available  to  use  in  the  simulations.  V&V  must  determine  that  the 
simulations  behave  according  to  the  reliability  data  or  predictions. 

V&  V  are  never-ending  processes.  The  tester  always  needs  to  know  if  the  ADS-enhanced  test 
environment  is  working  correctly.  For  complex  systems  of  systems  environments,  the  tester  must 
conduct  V&V  while  collecting  SUT  data  in  order  to  prove  post-test  that  valid  SUT  data  have 
been  collected.  Additionally,  the  ADS-enhanced  test  environment  may  not  exist  in  its  test 
configuration  until  the  first  test  event  is  conducted.  Also,  complex  test  environments  may  vary 
slightly  from  one  test  event  to  another  and  still  be  valid.  Conducting  some  form  of  V&V  during 
each  test  event  will  tell  the  tester  that  the  test  environment  is  valid. 


17 


5.0  Guide  to  the  W&A  of  an  ADS-Enhanced  Test  Environment 


As  stated  in  the  lessons  learned,  the  W&A  of  an  ADS-enhanced  test  environment  should 
proceed  hand-in-hand  with  the  design  and  development  of  the  environment.  The  JADS  Special 
Report— A  Test  Planning  Methodology-From  Concept  Development  Through  Test  Execution  at 
www.JADS.abq.com1  outlines  the  steps  in  planning  and  implementing  an  ADS-based  test.  The 
methodology  tailors  the  steps  given  in  the  HLA  FEDEP  model  to  test  and  evaluation  applications, 
resulting  in  an  ADS-enhanced  test  environment.  This  is  an  outstanding  guide  on  how  to  plan  an 
ADS-based  test  and  is  highly  recommended. 

This  guide  to  the  W&A  of  an  ADS-enhanced  test  environment  will  associate  the  appropriate 
W&A  activities  with  the  activities  contained  in  the  ADS-based  test  planning  and  implementation 
process. 

Included  in  the  test  planning  methodology  is  a  top-level  test  concept  development  process  that 
incorporates  decision  elements  associated  with  the  use  of  ADS  for  test  and  evaluation.  This  test 
concept  development  process  includes  Step  8,  Evaluate  Adequacy  of  the  Environment.  The 
purpose  of  this  step  is  to  satisfy  the  test  planners  that  the  conceptual  ADS-enhanced  test 
environment  is  rich  enough  in  terms  of  meaningful  interactions  to  support  a  sound  test.  This  can 
also  be  considered  as  early  conceptual  model  validation  and  further  supports  the  lesson  learned 
that  we  often  perform  V&V  and  don’t  realize  it.  In  fact,  by  documenting  the  process  the  tester 
could  partially  satisfy  the  requirement  to  perform  conceptual  model  validation. 

The  HLA  FEDEP  model  lists  six  steps  needed  to  develop  and  execute  an  ADS-based  test.  It 
should  be  noted  here  that  these  six  steps  might  involve  loop-backs  as  the  development  and 
execution  of  the  test  proceeds.  When  a  loop-back  occurs,  the  corresponding  V&V  will  be 
repeated  if  changes  are  made  to  the  design  or  execution  of  the  test.  The  six  steps,  as  listed  in  the 
test  planning  methodology,  are 

Step  1:  Define  Distributed  Test  Objectives 

Step  2:  Develop  Conceptual  Model 

Step  3:  Design  Distributed  Test 

Step  4:  Develop  Distributed  Test 

Step  5:  Integrate  and  Test  Architecture 

Step  6:  Execute  Distributed  Test  and  Analyze  Results 

An  ADS-based  test  has  the  requirement  that  the  ADS-enhanced  test  environment  and  its 
components  must  be  accredited  before  the  test  can  be  executed.  Since  the  accreditation  authority 
is  in  the  test  development  and  execution  chain,  a  step  needs  to  be  added  to  the  model  between 
step  5  and  step  6.  Step  5a  becomes  “Accredit  ADS-Enhanced  Test  Environment.” 


1  After  1  March  2001  refer  requests  to  Headquarters  Air  Force  Operational  Test  and  Evaluation  Center  History 
Office  (HQ  AFOTEC/HO),  8500  Gibson  Blvd.  SE,  Kirtland  Air  Force  Base,  New  Mexico  87117-5558,  or  SAIC 
Technical  Library,  2001  North  Beauregard  St.,  Suite  80,  Alexandria,  Virginia  22311. 
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5.1  Step  1.  Define  Distributed  Test  Objectives 

From  the  test  planning  methodology,  the  test  sponsor  or  evaluator  and  the  distributed  test 
development  team  define  and  agree  on  a  set  of  objectives  and  document  what  must  be 
accomplished  to  achieve  those  objectives.  This  is  a  test  planning  step  and  is  addressed  by  the 
test  planning  methodology.  (JADS  Special  Report — A  Test  Planning  Methodology-From 
Concept  Development  Through  Test  Execution) 

Early  in  this  step  the  test  manager  should  appoint  a  V&V  lead.  If  the  test  manager  is  the 
accreditation  authority,  also  appoint  an  accreditation  team  lead;  if  not  ask  the  accreditation 
authority  to  appoint  an  accreditation  team  lead  to  be  a  member  of  the  IPT  for  the  test. 

As  the  set  of  objectives  for  the  test  matures,  the  accreditation  team  lead  and  the  V&V  lead  should 
discuss  accreditation  requirements  and  how  the  V&V  will  be  conducted.  Broad  requirements  and 
acceptability  criteria  should  be  arrived  at  so  that  the  V&V  plan  can  be  developed.  Issues  such  as 
V&V  products  required  by  the  accreditation  team,  independent  V&V  versus  in-house  V&V,  and 
the  role  of  contractors,  if  any,  in  the  V&V  need  to  be  agreed  upon  and  documented.  It  is  also 
helpful  for  them  to  identify  SMEs,  to  be  used  throughout  the  test  and  V&V,  who  are  acceptable 
to  both  the  test  manager  and  the  accreditation  authority. 

The  accreditation  team  lead  also  needs  to  find  out  if  the  accreditation  authority  wants  an 
accreditation  team  from  within  the  accrediting  organization  (SIT  and  EW  Test)  or  a  team 
composed  of  subject  matter  experts  from  outside  the  organization  (ETE  Test).  Once  this  is 
accomplished,  write  an  accreditation  plan  describing  the  composition  of  the  accreditation  team, 
the  accreditation  objectives,  and  the  broad  accreditation  requirements  and  acceptability  criteria. 

It  is  at  this  point  that  the  overall  V&V  plan,  describing  the  tailored  process  that  will  be  used  by 
the  V&V  team  and  the  roles  of  the  members  of  the  IPT,  should  be  written.  This  plan  will  be 
nonspecific  with  regard  to  actual  V&V  activities  as  the  actual  simulations,  nodes,  requirements, 
and  acceptability  criteria  for  the  ADS-enhanced  test  environment  have  yet  to  be  determined.  It 
will  match  major  V&V  events  with  major  test  events  and  describe  the  specific  V&V  events  that 
will  be  conducted  to  meet  the  previously  agreed  upon  broad  acceptability  criteria.  It  will  also 
identify  future  V&V  plans  that  will  be  specific  in  nature.  This  plan  will  be  used  to  cost  the  V&V 
process. 

It  will  also  describe  in  some  detail  how  the  conceptual  model  will  be  validated  as  this  validation 
will  occur  prior  to  firm  requirements  being  developed.  Finally,  it  must  identify  the  types  of  data 
required  to  evaluate  how  well  the  ADS-enhanced  test  environment  meets  the  test  objectives. 
These  must  be  done  at  this  step  so  that  the  test  designer  can  make  provisions  for  the  collection  of 
these  data  in  the  test  design.  The  test  manager  must  approve  this  plan  after  review  by  the  test 
designer,  schedule  personnel,  resource  personnel,  and  the  accreditation  authority  or 
representative. 
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Early  validation  of  the  conceptual  model  may  begin  during  this  step  as  SMEs  are  employed  to 
review  the  federation  objectives,  approach,  and  technologies.  The  efforts  of  these  SMEs  should 
be  documented  for  future  use.  Any  work  that  tries  to  determine  how  well  the  ADS-enhanced  test 
environment  will  represent  the  real  environment  should  be  considered  as  conceptual  model 
validation. 

5.2  Step  2.  Develop  Conceptual  Model 

A  representation  of  the  real-world  domain  of  interest  is  developed  and  described  in  terms  of  a 
set  of  required  objects  and  interactions.  Most  of  the  activities  under  this  step  are  addressed  by 
the  test  planning  methodology.  ( JADS  Special  Report— A  Test  Planning  Methodology-From 
Concept  Development  Through  Test  Execution) 

This  step  is  composed  of  three  activities  that  are  very  important  to  the  W&A  of  an  ADS- 
enhanced  test  environment. 

5.2.1  Activity  2.1  -  Develop  Scenario 

The  basis  for  the  V&V  of  the  scenario(s)  to  be  used  in  the  test  must  be  established  here.  The 
V&V  team  must  work  closely  with  the  scenario  developers,  documenting  data  sources,  doctrinal 
sources,  and  fidelity  requirements.  The  requirements  for  the  V&V  of  the  scenario  including  the 
accrediting  authority  for  the  scenario  should  have  been  identified  in  the  V&V  plan  developed 
during  the  previous  step. 

5.2.2  Activity  2.2  -  Perform  Conceptual  Analysis 

At  this  point  that  the  distinction  among  test  team  members  and  V&V  team  members  becomes 
fuzzy.  The  purpose  of  this  activity,  for  the  test  team  members,  is  to  ensure  that  the  conceptual 
model  maps  to  the  objectives,  scenario,  doctrine  and  tactics,  and  fidelity  requirements  of  the  test. 
In  other  words,  determine  how  well  the  federation  represents  the  real-world  test  environment  and 
is  the  representation  adequate.  Do  this  in  order  to  reduce  risk  prior  to  proceeding  with  the 
development  of  the  federation.  The  V&V  team  wants  to  do  the  same  thing.  The  two  teams,  if 
they  are  composed  of  different  people,  should  work  together  within  the  IPT  structure  to  ensure 
that  this  activity  provides  the  required  degree  of  risk  reduction  to  the  test  manager,  as  this  is  its 
primary  purpose. 

With  respect  to  the  accreditation  of  the  ADS-enhanced  test  environment,  the  validation  of  the 
conceptual  model  is  a  freebie.  The  results  will  be  overshadowed  by  the  validation  of  the 
constructed  federation,  and  the  accreditation  decision  will  be  based  upon  the  performance  of  the 
federation,  not  a  conceptual  model.  As  stated  above,  conceptual  validation  is  conducted  in  order 
to  reduce  risk. 


20 


5.2.3  Activity  2.3  -  Develop  Distributed  Test  Requirements 

This  is  probably  the  most  important  activity,  with  respect  to  V&V,  in  the  test  planning 
methodology.  Requirements  to  the  person  conducting  V&V  are  like  specifications  to  a  person 
building  a  system.  They  dictate  what  the  federation  and  its  federates  have  to  do  to  satisfy  the 
test’s  objectives  and  measures.  As  stated  in  the  test  planning  methodology,  they  fall  into  several 
broad  categories.  These  categories  are  fidelity  requirements,  interaction  requirements,  latency 
requirements,  data  reliability  requirements,  and  data  analysis  requirements.  All  require  V&V 
effort  to  determine  how  well  the  federation  and  its  federates  meet  the  requirements. 

It  is  critical  that  the  V&V  team  be  a  part  of  the  IPT  and  participate  in  the  development  of  the 
requirements.  The  team  can  offer  valuable  insights  as  to  what  is  measurable  quantitatively  and 
what  must  be  measured  qualitatively.  It  is  equally  important  to  include  the  accreditation  team 
lead  who  has  the  responsibility  of  working  with  the  test  director  and  the  accreditation  authority  to 
develop  the  acceptability  criteria. 

If  the  requirements  state  what  the  federation  and  its  federates  should  do,  the  acceptability  criteria 
state  the  minimum  acceptable  performance  of  the  federation  and  its  federates.  Ideally,  the  tester 
would  like  for  any  simulation  of  a  system,  or  process,  to  be  perfect.  Practically,  close  is  often 
good  enough.  The  acceptability  criteria  define  close.  If  the  federation  and  its  federates  fail  to 
meet  acceptability  criteria,  the  accreditation  team  has  the  option  of  recommending  that  the 
federation  be  modified  to  meet  the  criteria  or  that  the  test  proceed  at  the  observed  level  of 
performance. 

Once  the  requirements  and  acceptability  criteria  are  identified,  the  detailed  V&V  plan(s)  may  be 
developed  stating  the  specific  V&V  activities  to  be  performed  during  the  design,  development, 
integration  and  testing  of  the  federation.  The  test  manager  must  approve  these  plans  after  review 
by  the  test  designer,  schedule  personnel,  resource  personnel,  and  the  accreditation  team. 

5.3  Step  3.  Design  Distributed  Test 

Distributed  test  participants  (federates)  are  determined,  and  required  functionalities  are 
allocated  to  the  participants.  (JADS  Special  Report— A  Test  Planning  Methodology-From 
Concept  Development  Through  Test  Execution) 

The  V&V  team  will  need  to  work  closely  with  the  designer  during  this  phase  as  it  functions  as  a 
quality  control  team  for  the  test  manager.  Its  responsibility  is  to  verify  that  HLA  federation  and 
federate  rules  have  been  followed,  where  applicable,  and  that  the  federation  and  federates 
conform  to  the  object  model  template.  It  should  also  document  any  deviations  from  the  HLA 
rules  and  the  reason  why  the  deviations  were  required. 

The  V&V  team  will  also  verify  that  the  detailed  design  will  meet  the  acceptability  criteria  arrived 
at  during  the  last  step  and  that  the  objectives  of  the  test  will  be  met  by  the  proposed  design.  The 
design  team  will  also  conduct  all  or  most  of  these  tasks.  The  V&V  team  provides  a  second  set  of 
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eyes,  with  a  slightly  different  perspective,  to  ensure  that  the  design  is  adequate  and  to  reduce  risk 
within  the  test.  The  use  of  modeling  tools,  to  model  the  federation  and  its  federates,  is 
recommended  as  a  V&V  or  design  tool  during  this  step. 

Finally,  the  V&V  team  will  verify  that  adequate  provisions  are  made  in  the  design  for  the 
collection  of  data  to  V&V  the  final  federation  as  assembled. 

5.4  Step  4.  Develop  Distributed  Test 

The  federation  object  model  (FOM)  is  developed  (if  HLA  is  implemented),  participant 
agreements  on  consistent  databases/algorithms  are  established,  and  modifications  to  federates 
are  implemented  (as  required).  (JADS  Special  Report — A  Test  Planning  Methodology-From 
Concept  Development  Through  Test  Execution) 

This  step  is  a  continuation  of  the  previous  step  and  the  V&V  effort  will  be  directed  toward 
reducing  risk.  As  federates  are  modified,  they  will  be  tested  to  ensure  they  meet  requirements  and 
that  they  continue  to  be  HLA  compliant. 

The  major  emphasis  at  this  point  should  be  on  ensuring  that  the  federation  will  satisfy  the  test’s 
requirements  and  objectives  and  less  on  standards  compliance.  The  purpose  of  V&V  is  not  to  act 
as  a  standards  policeman.  Given  the  choice  of  a  successful  test  meeting  all  requirements  and 
objectives,  or  blind  adherence  to  a  standard,  most  test  managers  will  decide  on  the  former. 
Deviations  from  standards  should  be  documented  and  reported  to  the  standards  authority  along 
with  the  “fixes”  employed  to  make  the  test  environment  work. 

As  they  are  identified,  components  of  the  federation,  to  include  hardware  items,  may  be  procured 
and  tested  for  HLA  compliance  and  functionality.  Items  such  as  routers,  crypto  devices,  and 
leased  communication  lines  may  also  be  tested  and  characterized  within  the  federation 
configuration.  It  is  recommended  that  a  test  bed  federation  with  dummy  federates  be  established 
using  all  of  the  actual  hardware  components  before  moving  to  the  next  step.  The  dummy 
federates  receive  and  publish  data  in  the  same  manner,  and  at  the  same  rate,  as  the  actual  federates 
are  expected  to  perform.  This  allows  the  test  bed  to  experience  the  same  interactions  and  data 
flow  as  the  actual  federation.  Interactions  and  data  flow  may  be  determined  from  the  model  of 
the  federation  recommended  in  the  previous  step. 

Using  the  test  bed,  hardware  upper  limits  can  be  established  for  data  flow  within  the  federation, 
and  the  effects  of  changes  to  the  architecture  and  data  flow  can  be  easily  observed.  Proper 
documentation  is  essential  at  this  point,  especially  if  changes  are  made  to  the  architecture  that 
deviates  from  the  HLA  standard. 

5.5  Step  5.  Integrate  and  Test  Architecture 

All  necessary  distributed  test  implementation  activities  are  performed,  and  testing  is  conducted 
to  ensure  interoperability  requirements  are  being  met.  {JADS  Special  Report — A  Test  Planning 
Methodology-From  Concept  Development  Through  Test  Execution) 
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Up  to  this  point,  the  purpose  of  the  V&V  conducted  was  to  reduce  risk.  The  V&V  conducted 
during  this  step  will  be  used  to  convince  the  accreditation  team  and  the  accrediting  authority  that 
the  ADS-enhanced  test  environment,  or  federation  if  HLA  is  used,  meets  the  acceptance  criteria 
or  at  least  performs  well  enough  to  proceed  with  the  test. 

V&V  activities  must  be  coordinated  with  the  integration  and  functionality  activities  conducted  by 
the  test  team.  Ideally,  no  additional  resource  requirements  will  be  placed  on  this  step  because  of 
V&V.  The  purpose  of  this  step  is  to  assemble  the  ADS-enhanced  test  environment  and  determine 
how  well  it  functions.  Functionality  is  measured  in  terms  of  how  well  the  environment  meets  its 
requirements.  V&V  also  need  to  determine  and  document  how  well  the  environment  functions. 
Whether  it  is  called  functionality  testing  or  V&V  is  immaterial,  both  have  the  same  objective. 

Compliance  issues  should  have  already  been  answered  and  documented.  That  is  why  the  use  of 
the  previously  mentioned  test  bed  is  recommended.  Deviations  from  HLA  standards,  in  order  to 
make  the  ADS-enhanced  test  environment  meet  requirements,  will  be  more  readily  accepted  than 
a  HLA-compliant  test  environment  that  does  not  meet  requirements. 

Occasionally,  a  system  or  system  of  systems  is  so  complex  that  the  ADS-enhanced  test 
environment  only  exists  during  actual  test  events.  An  example  was  the  JADS  ETE  Test  that 
contained  an  aircraft  flying  at  35000  feet  over  Texas.  Because  these  events  are  often  costly  in 
terms  of  time  and  resources,  the  test  manager  is  reluctant  to  allocate  one  of  these  events  to  V&V. 
Integration  and  functionality  testing  sufficient  to  satisfy  the  test  manager  that  the  ADS-enhanced 
test  environment  has  a  high  chance  of  success  during  an  actual  test  trial  can  take  place  on  the 
ground.  V&V  of  the  almost  real  ADS-enhanced  test  environment  can  also  be  conducted  to 
answer  how  well  the  environment  meets  the  test’s  requirements.  However,  if  the  test 
environment  exists  only  during  the  test,  the  V&V  conducted  are  again  risk  reduction  events. 

5.5.1  Step  5a.  Accredit  ADS-Enhanced  Test  Environment 

This  is  probably  the  easiest  step  in  the  process  provided  the  previous  W&A  activities  have  been 
performed  as  suggested.  You  normally  do  not  arrive  at  this  point  in  the  process  unless  the  ADS- 
enhanced  test  environment  is  performing  well  enough  to  satisfy  the  acceptability  criteria 
established  by  the  accreditation  authority.  In  addition,  if  you  have  fully  involved  the  members  of 
the  accreditation  team  in  planning  and  conducting  the  V&V  events,  they  already  know  how  well 
the  ADS-enhanced  test  environment  meets  the  acceptability  criteria.  This  is  especially  true  if  you 
provide  them  with  the  supporting  documentation  as  it  is  produced,  rather  than  dumping  all  of  it 
on  them  just  prior  to  the  accreditation  meeting. 

Obviously,  if  the  actual  test  environment  has  not  been  verified  and  validated,  as  described  in  the 
last  step,  the  accreditation  of  the  actual  test  environment  cannot  take  place  until  after  the  test 
event(s)  has  been  conducted.  The  accreditation  will  be  based  on  the  actual  test  performance  of 
the  environment  and  will  confirm  that  the  data  collected  during  the  test  are  valid.  The 
accreditation  team  should  meet  prior  to  the  test  and  consider  the  V&V  conducted  during  the 
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integration  and  testing  of  the  “near”  actual  test  environment.  It  can  still  recommend  proceeding 
to  test  with  the  stipulation  that  the  final  accreditation  will  not  occur  until  after  the  test. 

If  this  is  not  acceptable  the  accreditation  authority  will  have  to  be  willing  to  accept  the  allocation 
of  the  first  test  event  to  V&V  with  the  understanding  that  if  the  test  environment  is  subsequently 
accredited,  the  data  collected  during  the  V&V  are  valid  SUT  data. 

5.6  Step  6.  Execute  Distributed  Test  and  Analyze  Results 

The  distributed  test  is  executed,  outputs  are  generated,  and  results  provided.  (JADS  Special 
Report— A  Test  Planning  Methodology-From  Concept  Development  Through  Test  Execution) 

Why  V&V  after  the  ADS-enhanced  test  environment  has  already  been  accredited?  Well,  simply 
put,  most  ADS-enhanced  test  environments  are  so  complicated  that  the  only  way  you  can  tell  they 
are  functioning  correctly  is  to  repeat  some  of  the  V&V  activities  during  the  test  event.  In  order  to 
prove  that  you  have  collected  valid  test  data,  you  must  establish  that  the  ADS-enhanced  test 
environment  was  meeting  its  requirements. 
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6.0  Summary 

This  report  has  established  a  basis  for  the  W&A  of  an  ADS-enhanced  test  environment. 

It  has  done  so  by  reviewing  the  directives,  regulations,  and  formal  processes  that  apply  to  the 
W&A  of  an  ADS-enhanced  test  environment.  They  all  have  some  differences  in  detail,  but  they 
all  possess  the  common  thread  that  if  an  ADS-enhanced  test  environment  is  used  during  the 
testing  of  a  military  system,  it  must  first  undergo  W&A. 

Examples  of  the  W&A  processes  used  by  the  three  ADS-enhanced  tests  conducted  by  JADS 
were  presented  to  demonstrate  how  W&A  may  be  tailored  to  fit  the  particular  needs  of  a  test. 
The  three  tests  were  from  widely  different  test  domains,  used  different  ADS  technologies,  and  yet 
were  successfully  verified,  validated  and  accredited. 

W&A  lessons  learned  during  the  conduct  of  the  three  tests  were  presented.  The  two  most 
important  lessons  learned  were  that  V&V  are  affordable  if  performed  in  conjunction  with  test 
design,  integration,  and  functionality  testing  and  that  V&V  reduce  risk  for  the  tester.  If  done 
wisely,  V&V  can  provide  a  benefit  to  the  tester,  as  opposed  to  being  an  onerous  task  required  by 
regulation. 

The  last  section  of  this  report,  the  guide  to  conducting  V&V  of  an  ADS-enhanced  test 
environment,  is  purposefully  written  in  general  terms  to  emphasize  that  each  test  must  tailor  the 
specifics  of  the  W&A  process  to  its  needs  and  requirements.  What  is  critical  is  that  W&A  must 
start  at  the  beginning  of  the  test  process  and  the  people  involved  must  be  an  integral  part  of  the 
test  IPT.  Once  the  test  process  begins  areas  of  the  test  that  exhibit  the  highest  risk  should  be 
identified.  The  majority  of  the  W&A  effort  should  be  concentrated  in  those  areas. 

V&V  can  reduce  risk! 
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Annex  B:  Acronym  List 


ADS 

advanced  distributed  simulation 

AFEWES 

Air  Force  Electronic  Warfare  Evaluation  Simulator,  Fort  Worth,  Texas; 
Air  Force  managed  with  Lockheed  Martin  Corporation 

AFI 

Air  Force  instruction 

AR 

Army  regulation 

ATEWES 

Advanced  Tactical  Electronic  Warfare  Environment  Simulator 

C4ISR 

command,  control,  communications,  computers,  intelligence, 
surveillance  and  reconnaissance 

DAB 

Defense  Acquisition  Board 

DAP  AM 

Department  of  the  Army  pamphlet 

DIS 

distributed  interactive  simulation 

DMSO 

Defense  Modeling  and  Simulation  Organization,  Alexandria,  Virginia 

DoD 

Department  of  Defense 

DON 

Department  of  the  Navy 

DPRB 

Defense  Planning  and  Resources  Board 

DSM 

digital  system  model 

DT&E 

developmental  test  and  evaluation 

ETE 

JADS  End-to-End  Test 

EW 

JADS  Electronic  Warfare  Test 

EWIR 

electronic  warfare  integrated  reprogramming  database 

FAT 

federate  acceptance  test 

FEDEP 

federation  development  and  execution  process 

FIT 

federate  integration  test 

FOM 

federation  object  model 

GTRI 

Georgia  Tech  Research  Institute,  Atlanta,  Georgia 

HITL 

hardware-in-the-loop  (electronic  warfare  references) 

HLA 

high  level  architecture 

HWIL 

hardware-in-the-loop  (system  integration  references) 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

IPT 

integrated  product  team 

ISTF 

installed  systems  test  facility 

JADS 

Joint  Advanced  Distributed  Simulation,  Albuquerque,  New  Mexico 

Joint  STARS 

Joint  Surveillance  Target  Attack  Radar  System 

JROC 

Joint  Requirements  Oversight  Council 

JT&E 

joint  test  and  evaluation 

JTF 

joint  test  force 

M&S 

modeling  and  simulation 

MAIS 

Major  Automated  Information  System 

MDAPS 

major  defense  acquisition  programs 

MSEA 

modeling  and  simulation  executive  agent 

OAR 

open  air  range 

OSD 

Office  of  the  Secretary  of  Defense 
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OT&E 

operational  test  and  evaluation 

POM 

program  objectives  memorandum 

PPBS 

Planning,  Programming,  and  Budgeting  System 

RF 

radio  frequency 

RPG 

recommended  practices  guide 

SE 

synthetic  environment 

SECNAV 

Secretary  of  the  Navy 

SECNAVINST 

Secretary  of  the  Navy  instruction 

SIT 

JADS  System  Integration  Test 

SME 

subject  matter  experts 

SUT 

system  under  test 

T&E 

test  and  evaluation 

TCAC 

Test  Control  and  Analysis  Center  at  JADS,  Albuquerque,  New  Mexico 

USD  (A&T) 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 

V&V 

verification  and  validation 

W&A 

verification,  validation  and  accreditation 

VV&C 

verification,  validation  and  certification 
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